-
Notifications
You must be signed in to change notification settings - Fork 34
PLUGIN-1823: Retrying all SQLTransientExceptions #597
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: develop
Are you sure you want to change the base?
Conversation
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
7fcb0a0
to
9da75b8
Compare
9da75b8
to
ac813f0
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall, this looks everything is getting wrapped within Failsafe
where we might end up with having nested level retries, we need to ensure we add retries only where we are actually interacting with JDBC client and not top level functions.
For example adding retries to DriverManager.getConnection(connectionString, connectionProperties)
makes sense because you are actually interacting with the source db but adding retries to whole loadSchema(Connection connection, String query)
do not makes sense we need to be careful while adding such retries.
database-commons/src/main/java/io/cdap/plugin/db/action/DBRun.java
Outdated
Show resolved
Hide resolved
database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java
Outdated
Show resolved
Hide resolved
database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java
Outdated
Show resolved
Hide resolved
Please note E2E should not be modified and not fail with these changes. Otherwise, we have done something wrong which does not give expected failure messages. |
database-commons/src/main/java/io/cdap/plugin/db/sink/AbstractDBSink.java
Outdated
Show resolved
Hide resolved
database-commons/src/main/java/io/cdap/plugin/db/sink/AbstractDBSink.java
Outdated
Show resolved
Hide resolved
database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java
Outdated
Show resolved
Hide resolved
database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can see some level of duplication in both AbstractDBSource
& AbstractDBSink
, can we please move it to the common AbstractDBUtil
class?
database-commons/src/main/java/io/cdap/plugin/db/ConnectionConfig.java
Outdated
Show resolved
Hide resolved
cloudsql-mysql-plugin/src/test/java/io/cdap/plugin/cloudsql/mysql/CloudSQLMySQLSinkTest.java
Show resolved
Hide resolved
database-commons/src/main/java/io/cdap/plugin/db/connector/AbstractDBSpecificConnector.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
public final class RetryUtils {
public static Connection createConnectionWithRetry(RetryPolicy<?> retryPolicy, String connectionString,
Properties connectionProperties, String externalDocumentationLink) throws Exception {
try {
return Failsafe.with(retryPolicy).get(() ->
DriverManager.getConnection(connectionString, connectionProperties)
);
} catch (Exception e) {
throw unwrapFailsafeException(e, externalDocumentationLink);
}
}
public static Statement createStatementWithRetry(RetryPolicy<?> retryPolicy,
Connection connection, String externalDocumentationLink) throws Exception {
try {
return Failsafe.with(retryPolicy).get(connection::createStatement);
} catch (Exception e) {
throw unwrapFailsafeException(e, externalDocumentationLink);
}
}
public static PreparedStatement prepareStatementWithRetry(RetryPolicy<?> retryPolicy,
Connection connection,
String sqlQuery, String externalDocumentationLink) throws Exception {
try {
return Failsafe.with(retryPolicy).get(() ->
connection.prepareStatement(sqlQuery)
);
} catch (Exception e) {
throw unwrapFailsafeException(e, externalDocumentationLink);
}
}
public static ResultSet executeWithRetry(RetryPolicy<?> retryPolicy,
Connection connection,
String sqlQuery, String externalDocumentationLink) throws Exception {
try {
return Failsafe.with(retryPolicy).get(() -> connection.createStatement().executeQuery(sqlQuery));
} catch (Exception e) {
throw unwrapFailsafeException(e, externalDocumentationLink);
}
}
private static Exception unwrapFailsafeException(Exception e) {
if (e instanceof FailsafeException && e.getCause() instanceof Exception) {
if (e instanceOf SQLException) {
return programFailureException(e, externalDocumentationLink);
} else {
return (Exception) e.getCause();
}
}
return e;
}
private static ProgramFailureException programFailureException(SQLException e, String externalDocumentationLink) {
// wrap exception to ensure SQLException-child instances not exposed to contexts without jdbc
// driver in classpath
String errorMessage =
String.format("SQL Exception occurred: [Message='%s', SQLState='%s', ErrorCode='%s'].",
e.getMessage(), e.getSQLState(), e.getErrorCode());
String errorMessageWithDetails = String.format("Error occurred while trying to" +
" get schema from database." + "Error message: '%s'. Error code: '%s'. SQLState: '%s'", e.getMessage(),
e.getErrorCode(), e.getSQLState());
if (!Strings.isNullOrEmpty(externalDocumentationLink)) {
if (!errorMessage.endsWith(".")) {
errorMessage = errorMessage + ".";
}
errorMessage = String.format("%s For more details, see %s", errorMessage, externalDocumentationLink);
}
return ErrorUtils.getProgramFailureException(new ErrorCategory(ErrorCategory.ErrorCategoryEnum.PLUGIN),
errorMessage, errorMessageWithDetails, ErrorType.USER, false, ErrorCodeType.SQLSTATE, e.getSQLState(),
externalDocumentationLink, e);
}
}
You can create a RetryUtils
like above which accepts connection params.
database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java
Outdated
Show resolved
Hide resolved
Move retry logic into a separate class: RetryUtils and add exception handling
Refactored the code to add the retry logic only for the methods interacting with the JDBC client. |
database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java
Outdated
Show resolved
Hide resolved
database-commons/src/main/java/io/cdap/plugin/util/RetryUtils.java
Outdated
Show resolved
Hide resolved
@@ -12,7 +12,7 @@ errorMessageInvalidImportQuery=Import Query select must contain the string '$CON | |||
\ to 1. Include '$CONDITIONS' in the Import Query | |||
errorMessageBlankUsername=Username is required when password is given. | |||
errorMessageInvalidTableName=Error encountered while configuring the stage: 'Error occurred while trying to get schema from database.Error message: 'IO Error: Unknown host specified '. Error code: '17002'. SQLState: '08006'' | |||
errorMessageInvalidSinkDatabase=Error encountered while configuring the stage: 'Error occurred while trying to get schema from database.Error message: 'IO Error: Unknown host specified '. Error code: '17002'. SQLState: '08006'' | |||
errorMessageInvalidSinkDatabase=Error encountered while configuring the stage: 'Error occurred while trying to get schema from database. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not correct, why are we making the error message generic, it should cover the exact error message like before.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changes done. Updated the error messages, not including the dynamic/privacy data if present in error msgs.
@@ -1,4 +1,4 @@ | |||
errorMessageInvalidSourceDatabase=Error occurred while trying to get schema from database.Error message: 'Access denied for user ' | |||
errorMessageInvalidSourceDatabase=Error occurred while trying to get schema from database. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
similar comment here, it applies for all the error messages in the PR where we are reducing the exact messages and making it generic
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if the error messages are not coming like before, there is something wrong with the changes and we should fix there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changes done. Updated the error messages.
database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java
Outdated
Show resolved
Hide resolved
database-commons/src/main/java/io/cdap/plugin/db/sink/AbstractDBSink.java
Outdated
Show resolved
Hide resolved
…rn types for overridden methods to base class
protected String getExternalDocumentationLink() { | ||
return null; | ||
return "https://en.wikipedia.org/wiki/SQLSTATE"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this method still needed? can we remove it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not needed now. Removed. 7c8815d
protected String getExternalDocumentationLink() { | ||
return "https://en.wikipedia.org/wiki/SQLSTATE"; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this method still needed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, not needed now. Removed it. 7c8815d
…t longer used now.
@@ -87,13 +86,17 @@ public abstract class AbstractDBSource<T extends PluginConfig & DatabaseSourceCo | |||
Pattern.CASE_INSENSITIVE); | |||
private static final Pattern WHERE_CONDITIONS = Pattern.compile("\\s+where \\$conditions", | |||
Pattern.CASE_INSENSITIVE); | |||
private final RetryPolicy<?> retryPolicy; | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please remove empty line
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed 0b57ce6
71e3ddb
to
c6de31b
Compare
c6de31b
to
26efc64
Compare
0e9b88e
to
2284f2d
Compare
@@ -1,9 +1,9 @@ | |||
errorMessageInvalidSourceDatabase=SQL error while getting query schema: Error: Unknown database 'invalidDatabase', SQLState: 42000, ErrorCode: 1049 | |||
errorMessageInvalidSourceDatabase=errorMessage: SQL Exception occurred |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I still see generic error messages which is bad.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The earlier error message is changed as per the new code, and hence as per the latest code updated the error message not include the dynamic/privacy data present in error msgs here it contains.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we please add Open and Capture Logs
step to these scenarios :
-
database-plugins/mssql-plugin/src/e2e-test/features/mssql/mssql source/RunTimeWithMacros.feature
Line 217 in e5c5794
Scenario: Verify pipeline failure message in logs when user provides invalid Table name in importQuery of plugin with Macros -
database-plugins/mssql-plugin/src/e2e-test/features/mssql/mssql source/RunTimeWithMacros.feature
Line 258 in e5c5794
Scenario: Verify pipeline failure message in logs when user provides invalid Credentials for connection with Macros -
database-plugins/cloudsql-postgresql-plugin/src/e2e-test/features/source/RunTime.feature
Line 114 in e5c5794
Scenario: To verify pipeline failure message in logs when an invalid bounding query is provided -
database-plugins/cloudsql-postgresql-plugin/src/e2e-test/features/source/RunTime.feature
Line 156 in e5c5794
Scenario: To verify the pipeline fails while preview with invalid bounding query setting the split-By field
} else if (cause instanceof RuntimeException) { | ||
return (RuntimeException) cause; | ||
} else if (cause instanceof Error) { | ||
return new RuntimeException("Failsafe wrapped an Error", cause); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return new RuntimeException("Operation failed with error", cause);
} else if (cause instanceof Error) { | ||
return new RuntimeException("Failsafe wrapped an Error", cause); | ||
} else { | ||
return new RuntimeException("Failsafe wrapped a non-runtime exception", cause); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return new RuntimeException("Operation failed", cause);
Can we address these changes in other PR? We will create a new ticket to address these changes. Please confirm. |
I am not sure how to verify the e2e changes then? |
cbab045
to
1c0bfe5
Compare
PLUGIN-1823
Add Failsafe Retry poilcy to all the places in the database-plugins where
SQLTransientException
could be thrown.Added three new properties (hidden from UI)